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Evaluation of 
the NGSS Early 
Implementers Initiative 


The S. D. Bechtel, Jr. Foundation has commissioned WestEd's STEM Evaluation Unit 
to evaluate the NGSS Early Implementers Initiative in the eight participating public 
school districts. This independent evaluation is advised by a technical working group 
that includes representatives of the California Department of Education and the State 
Board of Education. Evaluators investigate three main aspects of the Initiative's NGSS 


implementation: 
districts’ local implementation, 
implementation support provided by the K-12 Alliance, and 


the resulting science teaching and leadership growth of teachers and adminis- 
trators, as well as student outcomes. 


In addition to this current Report #11, evaluators previously released: 


> The Needle Is Moving in California K-8 Science: Integration with English 
Language Arts, Integration of the Sciences, and Returning Science as a K-8 
Core Subject (Evaluation Report #1, October 2016) 


The Synergy of Science and English Language Arts: Means and Mutual 
Benefits of Integration (Evaluation Report #2, October 2017) 


Administrators Matter in NGSS Implementation: How School and District 
Leaders Are Making Science Happen (Evaluation Report #3, November 2017) 


Developing District Plans for NGSS Implementation: Preventing Detours 
and Finding Express Lanes on the Journey to Implement the New Science 
Standards (Evaluation Report #4, February 2018) 


Next Generation Science Standards in Practice: Tools and Processes Used by 
the California NGSS Early Implementers (May 2018) 


Making Middle School Science Whole: Transitioning to an Integrated 
Approach to Science Instruction (Evaluation Report #5, October 2018) 


Engaged and Learning Science: How Students Benefit from Next Generation 
Science Standards Teaching (Evaluation Report #6, November 2018) 
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Investing in Science Teacher Leadership: Strategies and Impacts in the NGSS 
Early Implementers Initiative (Evaluation Report #7, February 2019) 


Collaborative Lesson Studies: Powerful Professional Learning for 
Implementing the Next Generation Science Standards (Evaluation Report #8, 
September 2019) 


Environmental Instruction Catalyzes Standards-Based Science Teaching: 
How Environmental Literacy Aids Implementation of the NGSS (Evaluation 
Report #9, September 2019) 


Administrators Matter In NGSS Implementation (2019): Updated Findings on 
How School and District Leaders are Making Science Happen. (Evaluation 
Report #10, October 2019) 


California counties and school districts are 
implementing a critically needed change in how 
they evaluate science instructional materials 
before investing in local adoption. Past adoptions 
were often too superficial in nature, focusing on 
candidate materials’ overall look and feel, use of 
graphical elements, and availability of ancillary 
materials while insufficiently attending to the 
substance of the materials for high-quality teach- 
ing and learning. In contrast, the California NGSS 
Toolkit for Instructional Materials Evaluation (here- 
after referred to as TIME) process enables partic- 
ipants to use evidence-based measures to choose 
materials aligned to the Next Generation Science 
Standards (NGSS) that meet their district’s needs. 


This 11th report in the NGSS Early Implementers 
Initiative evaluation series is intended for school 
and district administrators, leaders of science 
professional learning, and state policymakers. It 
provides an overview of the full TIME process, 
including participants’ perceptions, a detailed 
description of the statewide TIME trainings of 
2018-19, and a vignette that illustrates a portion 
of the TIME process. 


The 2018-19 Statewide 
TIME Rollout 


When the California Department of Education 
released its approved list of 29 K—8 science 
instructional materials, district adoption commit- 
tees were primed for using a toolkit such as 

TIME to help them select NGSS-aligned instruc- 
tional materials to adopt. TIME trainings, which 


emphasized the labor-intensive Paper Screen 


step, occurred in three phases: (1) a master class 

to prepare 23 California science education lead- 
ers to lead large, regional TIME trainings; (2) two 
regional trainings, one in Northern California and 
one in Southern California, to which all county 
offices of education were invited to send a small 
team; and (3) dozens of local trainings facilitated by 


regional training participants. 


All but nine of California's 58 counties sent partic- 
ipants to a regional training. Local trainings were 
held in 39 of the 49 participating counties (almost 
80 percent) between January and October 2019. 


Challenges to 
Implementing TIME 


Participants detailed a variety of challenges asso- 
ciated with using the TIME toolkit as designed, 
most often citing the amount of time required. 
Although many local trainings closely followed 
the three-day training modeled in the master 
class and the regional trainings, evaluators also 
saw some modifications, including combining or 
rushing parts of the process. The report includes 
cautions from master class facilitators about 
which variations have the potential to reduce the 
integrity of a TIME training. 


vi 


It’s About TIME: A Rigorous New Process for 
Selecting Instructional Materials for Science 


> Consider the TIME approach as a support for 
Recommendations learning about the NGSS. 


The report recommends that adoption > Know that even after adopting NGSS instruc- 
tional materials, teachers and administrators 
will continue to need professional learning. 


committees: 


> Identify the unique needs and priorities of 
their district that they hope instructional 
materials will help them address. 


> Acknowledge the benefit of rigorously review- 
ing candidate instructional materials in all 
content areas. 


> Consider taking advantage of TIME, an 
existing tool developed specifically to guide 
the review of NGSS-aligned instructional 
materials. 


SM PO BP OR vii 


California counties and school districts are imple- 
menting a critically needed change in how they 
evaluate candidate science instructional materials 


for local adoption. 


Historically, districts have expended relatively 
little effort in selecting instructional materials 

to adopt (BSCS Science Learning, 2020; Landes et 
al., 2004). Some localities have simply conducted 
“thumb tests” — flipping through a textbook’'s 
pages to see what they notice at a glance — or 
other sorts of cursory reviews. Other common 
selection processes involve using mostly super- 
ficial evaluation criteria, such as appraising the 
quantity, look, and feel of graphics and pictures or 
looking for the presence of various ancillary mate- 


rials, such as handouts or assessment banks. 


However, given the substantial cost of mate- 
rials and their significant impact on teaching 
and learning, there is a need for a more rigorous 
process for inspecting candidate materials. This 
need is particularly pressing for science educa- 
tion in California, as teachers need high-quality 
materials that are aligned to the California Next 
Generation Science Standards (hereafter referred 
to as the NGSS), which require substantial shifts 
in teaching and learning practices from prior 


science standards. 


This process was very beneficial for our team. It was 
an opportunity to calibrate and have a shared vision for 
reviewing [science] instructional materials next year. | 


am mentally exhausted, but excited and motivated for 
this process! 
—School district participant in a local training on the 


California Next Generation Science Standards Toolkit 
for Instructional Materials Evaluation 


Background 


A more rigorous and intensive process has been 
developed to enable educators to deeply analyze 
candidate science resources and determine 

their suitability for the kinds of science learn- 

ing called for by the NGSS. The NextGen Toolkit 
for Instructional Materials Evaluation (NextGen 
TIME)’ is a suite of tools and processes for evaluat- 
ing and selecting instructional materials aligned 
with the NGSS (NGSS Lead States, 2013). Based on 
the original work of the K-12 Alliance at WestEd 
and developed for national application by BSCS, 
Achieve, and the K-12 Alliance, NextGen TIME 
provides a detailed process for curriculum adop- 
tion committees to collaboratively analyze mate- 
rials based on key criteria and scoring guidance. 
Furthermore, NextGen TIME provides educators 
with a transformative professional learning expe- 
rience highly focused on the NGSS. 


1 NextGen TIME is a new toolkit, but is rooted in processes developed over the last two decades. In the early 2000s, BSCS 


materials (Landes et al., 2004). 


and the K-12 Alliance at WestEd collaborated to create a rigorous process and a guidebook for selecting science instructional 
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The California NGSS Collaborative? customized 
the national NextGen TIME toolkit to align it with 
the state's version of the NGSS (Henriques et al., 
2019), including the state's focus on environmental 
literacy (Nilsen et al., 2019). This report focuses 
only on California's version of the toolkit, CA NGSS 
TIME (hereafter referred to as TIME). 


How TIME Connects to 
the Early Implementers 
Initiative 

This report is included in the Early Implementers 
evaluation report series because most of the 

Early Implementer districts have implemented 
TIME to some extent, and evaluators have 

been able to collect additional data about TIME 
from these districts. Further, four of the K-12 
Alliance Regional Directors — who have provided 
professional learning and support to the Early 
Implementer districts for the duration of the Early 
Implementers Initiative (the Initiative) — contrib- 
uted to the development of the TIME toolkit 

and led the initial TIME “master class” training 

of trainers in November 2018. Attendees of the 
master class, including five Project Directors 

and four Core Teacher Leaders from the Early 
Implementer districts, went on to lead the TIME 
regional trainings for county office of education 
personnel in December 2018. In addition, two 
Regional Directors facilitated local TIME train- 
ings, which took place in 2019 at county offices of 
education and school districts, including the Early 


Implementer districts. 


Of the eight Early Implementer districts, five 
districts have used or plan to use TIME to evalu- 
ate candidate instructional materials for district 


adoption. The remaining three Early Implementer 
districts have varied reasons for not using the 
TIME approach. One large district is choosing not 
to adopt instructional materials because it would 
require an initial cost investment with ongoing 
subscription costs. Another district is content, for 
the time being, with the materials it developed 
over the course of the Initiative and is working to 
share them with all of its K-8 teachers. The third 
district had already chosen curriculum before the 
TIME trainings were offered. 


About This Report 


This report describes the TIME process and toolkit 
and how they have been shared with and received 
by science educators throughout the state of 
California. The report also describes how the 
TIME approach and review of instructional mate- 
rials may be beneficial to districts in their NGSS 


implementation efforts. 


Although this report describes a toolkit and 
process for selecting NGSS-aligned science 
instructional materials, it does not provide 
specific recommendations or suggestions for 
which instructional materials to select or adopt 
because through participating in the TIME 
process, district teams collect evidence and 
come to consensus on instructional materials 
that meet their unique needs, which can differ 
across districts. 


2 Members of the California NGSS Collaborative include the California Department of Education, the California Science 
Project, the California Science Teachers Association, local county offices of education, theCalifornia County Superintendents 
Educational Services Association and its Curriculum Instruction Science Committee, and the K-12 Alliance at WestEd. 
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Data Sources 


This report draws upon a variety of data sources 
from three phases of TIME trainings — the 
master class, regional trainings, and local 
trainings — as well as from interviews of Early 
Implementer leaders. Specifically, the data sources 


include: 


> Extensive interviews with 20 facilitators and 
26 attendees of TIME trainings 


> Observations of the TIME master class, both 
the north and the south regional TIME train- 
ings, and 15 local TIME sessions around the 
state 


> Survey responses from 178 attendees of 
regional TIME trainings 


> Survey responses from 41 facilitators and 425 
attendees of local TIME trainings 


> Interviews in early 2020 with eight Early 
Implementer Project Directors and three Early 
Implementer Regional Directors 


> Focus group with three master class 
facilitators 


Participants and phases of data collection are 
described in more detail in Appendix A. 
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Purpose of TIME 


TIME is a comprehensive and rigorous process for 


evaluating how well instructional materials: 


> Align to the NGSS, including whether they 
include phenomena/design problems, the three 
dimensions, the Environmental Principles and 
Concepts, and a logical sequence. 


> Provide learning experiences corresponding to 
desired student learning outcomes and useful 
teacher supports and resources. 


> Are consistent with district needs and priori- 
ties, which are determined in the first activity 
in the TIME process. 


In addition, TIME was designed to be a power- 

ful NGSS professional learning experience. For 
example, TIME teaches participants? to recog- 
nize the conceptual shifts required by the NGSS 
and provides hands-on practice locating specific 
evidence of NGSS features, including phenomena/ 
design problems and the three dimensions, in 


instructional materials. 


Although TIME was designed for evaluating 


science instructional materials, districts could use 
a process like TIME when selecting instructional 


materials in any content area. 


How Is the Process 
Structured? 


TIME is a comprehensive, six-step, evidence-based 
process (see Figure 1) for evaluating and adopting 
NGSS-aligned instructional materials that will 

fit a district’s identified needs and priorities. The 
TIME toolkit offers a detailed procedure, with 
supporting tools (e.g., Score Sheets, Strengths and 
Limitations charts) that guide users through the 


six steps, which are summarized in this section. 


3 The term participants refers to individuals participating in the TIME process, which is a collaborative group activity. 
Participants typically include county office of education professional learning providers and district-level adoption committee 


members. 
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Figure 1. The six steps of NGSS TIME 


Develop 
District Lens 


Prescreen 


Paper Screen 


Select and Pilot 


Recommend 


Materials 


Source: Graphic from handout included in binder provided at TIME trainings. 


Step 1: Develop District Lens 


During this step, participants collaboratively 
identify their district’s unique needs, resources, 
and instructional priorities. The district lens can 
be shaped by school demographics, as well as by 
school or district instructional priorities (e.g., a 
focus on technology integration). This step prepares 
the team to evaluate instructional materials based 
on the district's unique needs, which assists the 
adoption committee in selecting the best possible 
materials for their particular teacher and student 
populations. (Typical duration: 3-4 hours)* 


Step 2: Prescreen 


This step is designed to efficiently narrow down a 
relatively large number of available science instruc- 
tional materials to just a few of the most promising 
options, which will undergo a more thorough and 
time-intensive review in the next step, the Paper 
Screen. During the Prescreen, participants examine 
multiple sets of instructional materials for align- 
ment with the NGSS. An essential component of 
this step is a professional learning session that 
calibrates the team and models the Prescreen 
process to be used for instructional materials 


4 Typical duration for each step comes from an informational two-page handout disseminated at the TIME trainings. 
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under consideration. (Typical duration: 16 hours, 


including 6-8 hours of training) 


Step 3: Paper Screen 


This is the most time-intensive and potentially 
most important step, which is why it was the 
focus of the TIME trainings delivered through- 
out California in 2019. Working with one set at 

a time of materials that “passed” the Prescreen, 
participants apply a series of rubrics to evaluate 
evidence of features of high-quality, NGSS-aligned 
science instructional materials.’ They individually 
score the materials using several criteria in the 
rubrics and engage in consensus-building activ- > 
ities to cooperatively determine which materials 
merit advancing to the Pilot Materials step. As in 
step 2 (Prescreen), an essential component of the 
Paper Screen step is for the adoption committee 
to engage in a shared professional learning expe- 
rience in order to calibrate themselves using a 
sample set of instructional materials not under 
consideration for adoption. (Typical duration: 24 
hours of training, plus approximately 16 hours per 


set of materials under consideration) 


The following rubrics are used to evaluate the 
materials during the Paper Screen step: 


> Rubric 1 - Foundations: Do the instructional 
materials include the basics for NGSS align- 
ment (i.e., phenomena/problems; the three 
dimensions; Environmental Principles and 
Concepts [EP&Cs], if applicable; and a logical 
sequence)? 


> Rubric 2 — Student Work: How are students 
going to learn? For example, do students 


have opportunities to explain phenomena/ 
problems or build a three-dimensional concep- 
tual framework? Do the materials leverage 
students’ prior knowledge and experiences, 
provide learning experiences that develop 
metacognition, and provide equitable learning 
experiences for all students? 


Rubric 3 — Monitoring Student Progress: 

How are students assessed? For example, 

are the assessments designed to monitor 
three-dimensional learning and EP&Cs? Will 
they capture student progress over time? Do 
the assessments include self- or peer-assess- 
ments and a variety of measures, support, and 
strategies for ensuring equitable access? 


Rubric 4 — Teacher Support: How do the 
instructional materials help teachers facili- 
tate students’ understanding of phenomena/ 
problem solutions using the three dimensions? 
For example, do the teacher materials provide 
guidance for supporting coherence, effective 
teaching, students with diverse learning needs, 
and phenomena-driven and problem-driven 
three-dimensional learning? Do the materi- 
als provide support for teachers to monitor 
student progress? 


Rubric 5 — Program Evaluation: Is high quality 
consistent across a grade level or between 
grade level progressions? (Optional)® 


Each rubric is composed of three to five criteria 
that are examined by participants one at a time. 
For summaries of the criteria in the rubrics, see 


Appendix B. 


The vignette at the end of this section provides a 


more detailed description of the Paper Screen step. 


5 Participants examine only a section of the materials rather than the full set. Further, it is recommended that, across the 
candidate instructional materials being evaluated, sections with similar content be chosen for the Paper Screen step to facilitate 


comparison of materials. 


6 Agroup may choose to complete rubric 5 between the Paper Screen and Pilot Materials steps to see if high quality is 
consistent across a grade level (e.g., across modules, chapters, units) or between grade level progressions. 
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Step 4: Pilot Materials 


This step allows teachers to use in the classroom 
the instructional materials chosen in the Paper 
Screen. The pilot-testing teachers collect addi- 
tional evidence while using the instructional 
materials during their instruction. This allows a 
more thorough analysis of each set of instructional 
materials under review and allows for additional 
evidence from teachers and students to be used in 
step 5, Select and Recommend. (Typical duration: 


varies) 


Step 5: Select and Recommend 


This step provides a decision-making framework 
to support the adoption committee in coming 

to consensus about the instructional materials 

to be recommended for adoption. Evidence and 
data from steps 1-4 are used as support for deci- 
sions, as this district level participant stated in an 


interview: 


My job is to present to [our school board] 
that we're recommending [instructional 
materials] because of what we found, 
based off of the evidence that it has all the 
components that we need for it to be the 
best program out there. (School district 


participant in a local TIME training) 


In some cases, districts have decided that none of 
the materials they reviewed were a fit: 


And then after using the TIME process, 
we realized that the materials that were 
out there were not actually in line with 
those components that I mentioned 
earlier: phenomena driven, prior knowl- 
edge, access, and equity. And so [the 


adoption committee] decided not to adopt 


anything. (Early Implementer Project 
Director who co-facilitated TIME) 


(Typical duration: varies) 


Step 6: Implement 


In this final step, the toolkit provides resources 
for planning and monitoring ongoing instruc- 
tional material implementation. It is important to 
note that Early Implementer district leaders have 
acknowledged that no instructional materials 
will be a perfect fit for any district. Consequently, 
many, such as the following Project Director, have 
stated that they intend to modify their district's 
adopted instructional materials or supplement 
them: 


What we found [in the instructional 
materials we reviewed] were investiga- 
tions that needed to be supplemented to 
make more of a hands-on experience in 
the classroom that could connect to the 
program and the phenomena that was 
driving the learning. We needed more 
to engage the students and keep them 
going in helping them to understand with 
the kinesthetic component of learning. 
(Early Implementer Project Director who 
co-facilitated TIME) 


(Typical duration: length of adoption) 


It is important to emphasize that, as described 
above, training is required before engaging in 
steps 2 and 3, to calibrate the evaluation of criteria 
and to familiarize adoption committee members 
with the process of looking for evidence before 
beginning to deeply examine candidate instruc- 


tional materials. 
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This vignette is representative of a typical TIME 


Paper Screen experience. The aspects of the Paper 
Screen process described below were commonly 


observed by evaluators. 


Three county office of education facilitators met 
with 30 participants from the same school district 
for a three-day TIME training on the Paper Screen 
process. The participants were divided into five 
chapter-specific groups to correspond to the five 
chapters of the instructional materials they would 
be reviewing.’ Their first assignment was to get 
acquainted with their chapter and create a graphic 
representation of the storyline of their chapter on 
a poster. The posters were placed on the wall in 
sequence. A presenter from each chapter group 
walked everyone through their chapter so that the 
whole group understood, and could see by looking at 
the posters on the wall, the storyline of the entire 


set of instructional materials from start to finish. 


Participants then used Rubric 1 from the toolkit 

to begin to further review their chapters. They 
examined the materials for evidence of the first 
important feature of high-quality NGSS instruc- 
tional materials, phenomena/problems used to 
drive student learning. When evidence of this 
feature was found, a pink sticky note was placed in 
the appropriate location on the chapter storyline. 
Then each chapter group presented what they 

had found. Members of other chapter groups had 


a chance to ask clarifying questions to make sure 
they agreed that the colored sticky on the poster 
was in fact evidence that phenomena/problems 
were present. They repeated this process for the 
other criteria in Rubric 1 (i.e., presence of the three 
dimensions, EP&Cs, and a logical sequence). 


When evidence of all of the rubric's criteria had 
been marked with colored stickies on the chap- 
ter posters, it was time to score the entire set 

of instructional materials. First, individuals 
recorded their own score (1, 3, or 5)° for each crite- 
rion in Rubric 1 in their notebooks. Then the whole 
group stood in a circle and shared their scores. A 
facilitator asked everyone, at the count of three, 
to show their score by holding up the appropriate 
number of fingers. The scores were close, but not 
unanimous. Most of the group had raised three 
fingers, but two people raised five. The facilita- 
tor recorded these scores on the rubric poster. 
Some participants explained the reasons for their 
scores. Occasionally, they scored based on an 
inference rather than evidence from the chapters. 
For example, one person noted that a phenomenon 
was present in multiple places in the unit. When 
asked to describe where he found evidence of the 
phenomenon, he was unable to document it in the 
text. He had inferred that a teacher would be able 
to use the question posed to students in the unit 
to identify a related phenomenon. Facilitators 
reminded the group that TIME requires that 


7 Participants used a set of instructional materials called Disruptions in Ecosystems for the Paper Screen process. Disruptions 
in Ecosystems was chosen for training purposes because it is recognized as being mostly aligned to the NGSS, and it is 

not one of the 29 sets of K—8 instructional materials approved by the California State Board of Education. The full list of 
approved materials is posted on the California Department of Education's (2018) website: https://www.cde.ca.gov/ci/sc/im/ 


adoptedsciprograms2018.asp. 


8 Participants used a scoring scale to indicate their assessment of the quality of each criterion. On this scale, 1 = low quality, 


3 = medium quality, and 5 = high quality (see Appendix B for score descriptions across all five rubrics). 
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they focus on what is evident in the instructional 
materials, not how teachers might modify them. 
After this share, facilitators asked participants 
to score the criteria again. This time all raised 
three fingers. They took a break and repeated the 
process with the next criterion, the three dimen- 
sions of the NGSS. 


After the set of instructional materials was 
assigned a consensus score for each criterion, 
the whole group discussed and assigned an over- 
all score based on the entire rubric. Because the 
materials had earned a score of 3 on most of the 
criteria, the group easily decided that the overall 
Rubric 1 score should be 3 (see Figure 2). 


Figure 2. Poster with recorded initial and 
final Rubric 1 scores 


It had been a productive day, with a combination of 
sitting and moving about and plenty of animated 
discourse. Even participants who arrived with 
deep knowledge of the NGSS remarked that they 
were learning even more about the standards. 


They would start with Rubric 2 in the morning. 


Over the next two days, the process was repeated 
with the three remaining rubrics. On the second 
day, the whole group completed Rubric 2 and 
began Rubric 3. They finished Rubrics 3 and 4 on 
the last day and unanimously determined that the 
instructional materials merited pilot testing. 


This vignette illustrates how thoroughly instruc- 
tional materials are examined by participants 
during the TIME Paper Screen step. It is important 
to note that the whole group focused carefully 

on one criterion at a time. The group members 
also documented the evidence of the criteria they 
found, and they engaged in consensus building for 


each criterion score. 


In the event that a set of candidate instructional 
materials receives consistently low scores on 
Rubric 1 (which was not the case in the vignette 
above), it would not be reviewed further (i.e., 
participants would not use Rubrics 2-5 to 
continue to examine the materials). However, 
when instructional materials show promise 

for adoption based on Rubric 1 scores, partici- 
pants continue to the other toolkit rubrics and 
complete the Paper Screen process. Occasionally, 
consensus is not reached after a couple of 
rounds of sharing scores, and facilitators may 
ask if those in the minority (i.e., those whose 
scores had the fewest tallies) can “live with” the 
final score from the majority; this usually only 


occurs at the criterion level. 


O 


As described above, TIME is a rigorous, multistep, 


multiday process for reviewing and selecting 
instructional materials. TIME training attend- 
ees and facilitators noted several benefits of the 
TIME approach and described the effectiveness of 
the TIME experience. Participants also reported 
challenges to implementing TIME, and evaluators 
observed variations to implementing the Paper 


Screen model as intended. 


Benefits of the TIME 
Approach 


TIME training attendees and facilitators reported 
that they found value in the TIME approach for 
instructional materials adoption. They believe 
that TIME is useful in a variety of ways, first and 
foremost for evaluating the quality of instruc- 
tional materials based on evidence and for provid- 


ing professional learning about the NGSS. 


Is Useful for Materials Adoption 


Participants reported that the TIME toolkit 
enabled them to conduct a thorough and objective 
review of instructional materials. Some inter- 
viewees described how the TIME process is a more 
rigorous approach to materials adoption compared 
with past approaches because it is evidence driven 


and not just based on opinion. 


As far as looking at instructional materi- 
als, it goes back to being very intentional 
with what you're looking at and how 
you're assessing it and making sure that 
it's evidence based. Not just what you 
like, what you don't like, or what you think 
will work based on your own experience, 
but really looking at data to make sure 
that the kids are learning what they're 
supposed to. (School district participant 
in a local TIME training) 


Although TIME is designed for science materials 
adoption, survey respondents indicated that the 
TIME process was useful for adopting instruc- 
tional materials in science and other subjects. 
When asked how useful they thought the TIME 
approach was for materials adoption in science, 
a large majority of regional training attendees 
(91 percent) thought that the approach was "very 
useful” (64 percent) or “useful” (27 percent) (see 
Figure 3). Similarly, 92 percent of local training 
attendees reported that it was “very useful” (64 
percent) or “useful” (28 percent). 
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Figure 3. Participants’ perceptions of the usefulness of the TIME approach for materials 
adoption in science 


How useful do you think the TIME approach is for materials adoption in science? 
100 
80 
60 


40 


20 


: Not Useful Somewhat Useful Very Useful 
Useful 
@ Attendees of regional trainings @ Attendees of local trainings 


Sources: CA NGSS TIME Survey for Regional Training Participants, administered by WestEd in 2019 (n = 174) and 
CA NGSS TIME Survey for Local Training Participants, administered by WestEd in 2019 (n = 381). 


In addition, the majority of both regional and we wouldn't have selected the curriculum 
local training attendees reported that the TIME that we did. (School district facilitator in a 
approach was “very useful” or “useful” for mate- local TIME training) 

rials adoption in content areas other than science 


88 tand 8 t, tively). Sree ; 
ar Se se ae Many of our districts did not go through 


Some training attendees expressed the view that a particularly robust process to choose 
the TIME process is worth the investment of instructional materials [for math]. And 
staff time and effort in order to ensure they are there is an immense amount of buyer's 


satisfied with the instructional materials that 1 : F Batueer 
remorse that's being voiced publicly in our 


they choose. 
- county right now. (County office of education 


Wouhnveshadutedchersawhouhaveabeen participant in a regional TIME training) 


doing the [TIME] process say that they 
would like to see something similar done Helps With Understanding the NGSS 
with future [materials] adoptions and 


that if we had had this process for math, The TIME experience was perceived to be useful 


for learning about the NGSS in general and the 


i 
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three dimensions in particular. This perception 
was true for participants with strong as well as 
weak background knowledge of the standards. 
Some participants reported that TIME trainings 
can serve as valuable professional learning about 
the NGSS, and the knowledge gained can be 
applied in the classroom and when evaluating and 
selecting candidate instructional materials. 


This training was very intense, but one of 
the best trainings I have attended. I feel 
very empowered to work with my district 
and use this tool to make the best selec- 
tion for our science adoption. In addition, 
my knowledge and understanding of the 
NGSS has greatly improved because of 
this specific training. (School district 
participant in a local TIME training) 


We had some teachers who are much 
deeper into the NGSS and have a greater 
understanding of the classroom applica- 


tion and things, and then we have some 


understandings of the framework as well. 
The training may be designed to select 
instructional materials, but the process 
has helped build a core of NGSS ambassa- 
dors from across our district and primed 
us for implementation. (School district 


facilitator in a local TIME training) 


It really helped us dig into the standards 
and the three dimensions and how they 
work. We've been to trainings before 
about NGSS, but it kind of had just been 
an overview of how they're structured. 
And unless you're looking at [the NGSS] 
every single day and digging into them, 
you're not getting that depth of under- 
standing. The whole process really gave 
us a deeper understanding of the NGSS 
and how they're supposed to function in 
the classroom. (School district partici- 


pant in a local TIME training) 


more novice teachers and administrators The majority of survey respondents felt that their 
there. They all felt valued and comfort- experience using TIME “greatly strengthened” or 
able. They all felt pushed. (School district 


participant in a local TIME training) 


“strengthened” their ability to recognize evidence 
of NGSS aspects in science instructional mate- 
rials (see Figure 4). This was true for all seven of 
the NGSS aspects listed in the survey, although 
The TIME training has been the quickest the lowest percentage of respondents indicated 
learning experience that I have observed that TIME “greatly strengthened” their ability to 


for people new to the NGSS, and [it] recognize evidence of engineering design. 


pushed our NGSS experts to realize new 
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Figure 4. Extent to which TIME strengthened participants’ ability to recognize evidence of 
NGSS aspects in science instructional materials 


How Phenomena are used to engage students’ prior knowledge 8% | 47% 47% 
3% 
The Science and Engineering Practices (SEPs) 10% 45% 45% 
2% 
The Crosscutting Concepts (CCCs) 10% 46% 46% 
3% 
The Disciplinary Core Ideas (DCls) 11% 46% 46% 
2% 
How the 3 dimensions of the NGSS (i.e., SEPs, CCCs, and DCls) are 10% 44% 44% 
used together to impact student... 2% 
The Environmental Principles and Concepts (EP&Cs) 15% 46% 46% 
7% 
Engineering Design 22% 45% 45% 
0 20% 40% 60% 80% 100% 
@ Greatly Strengthened @ Strengthened 
@ Slightly Strengthened @ Did not strengthen 


Source: CA NGSS TIME Survey for Local Training Participants, administered by WestEd in 2019 (n = 334). 


Helps With Under. standing Features of Training participants answered questions about 
High-Quality Instructional Materials the effectiveness of the TIME experience in achiev- 


ing two goals (see Figure 5). Views were gener- 


EOE GHEE NEIQ ON 2] HSE ISS EE ally positive, with more than half of participants 


that using TIME improved participants’ under- reporting that the TIME experience was “very 


standing of characteristics of high-quality, NGSS- effective’ in meeting each of these \goale. Very 


aligned instructional materials. Also, participants 
noted that using the TIME rubrics enabled them to 


few respondents (1 to 2 percent) reported that the 


TIME experience was “not effective” in achieving 


recognize elements of what students should learn either of these goals. 


and provided a shared vocabulary for reviewing 


instructional materials. 
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Figure 5. Participants’ perceptions of the effectiveness of the TIME experience in achieving 


two goals 


Improved your understanding of the characteristics of high-quality 
instructional materials for teaching the NGSS 


Increased your understanding of the importance of rigorous 
instructional materials selection and adoption 


0 20% 40% 60% 80% 100% 
@ Very Effective @ Effective 
@ Somewhat Effective @ Not effective 


Source: CA NGSS TIME Survey for Local Training Participants, administered by WestEd in 2019 (n = 334). 


One Project Director described how teachers in I love the tools. I love the rubrics. I think 


their district have been encouraged to use newly that, used correctly, they actually give you 


adopted instructional materials in science lesson : eer ; 
an amazing amount of insight into what 


studies as a way to learn the new materials: ; ; : 
students will be learning, and that's what 
we wanted. (School district participant in 


a local TIME training) 


One of the options with lesson study this 
year is that teams get to, or can opt to, use 


the recommended material or pilot mate- 


rial. So they get to actually experience [the EME enhanced my ability: te recogiize 


: : critical aspects of quality instructional 
recommended material] before we push it P y quality 


F : materials and gave our district a common 
out and drop it on their doorstep. g 


language and experience for the process. 
Participants discussed other things they valued (School district participant in a local 
about the TIME toolkit, including the rubrics and TIME training) 
how engaging in the collaborative TIME process 


led to a shared understanding of instructional 


priorities among district teams. 
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An effort to implement TIME statewide in 


California began in late 2018 and continued 
through the fall of 2019. With the NGSS being 
more widely implemented in districts than in 
previous years and a list of science instructional 
materials having just been approved by the 
California Department of Education (2018), it was 
an opportune time to promote the use of this 
toolkit for evaluating and selecting high-quality, 
NGSS-aligned instructional materials. 


To help foster statewide implementation, TIME 
trainings occurred in three phases, which are 
described below. The primary purpose of all 
three training phases was to model step 3 of the 
TIME process, the Paper Screen. Although the 
Paper Screen is just one of six steps, it is the most 
time-intensive, the most comprehensive, and 


arguably the most important step. 


Master Class 


In November 2018, a TIME master class was held 
to prepare 23 California science education leaders 
to lead large, regional TIME trainings the follow- 
ing month. The five master class trainers included 
staff from BSCS, Achieve, and the K-12 Alliance. 
The 23 attendees included representatives from 
the California Science Project; the NGSS Early 
Implementer districts; the K-12 Alliance; insti- 
tutions of higher education; and school districts, 
including five district Project Directors and four 
Core Teacher Leaders from Early Implementer 


districts. The group spent three days briefly learn- 
ing about the six-step TIME toolkit, but focused on 
how to facilitate step 3, the Paper Screen. A set of 
sample instructional materials that was not on the 
California Department of Education list of 2018 
approved Science Publishers and Programs? was 
used for training purposes. 


Regional Trainings 


In December 2018, these 23 leaders facilitated 
two regional TIME trainings, one in Northern 
California (Sacramento) and one in Southern 
California (Claremont). A small number of Early 
Implementer district leaders, who had learned the 
TIME process from previous Early Implementer 
professional learning, also facilitated these 
regional trainings. The regional trainings were 
open to any individuals responsible for supporting 
districts with the adoption of science materials.’° 
Materials were provided to attendees so that they 
could return and conduct local TIME trainings 
with their constituents. Forty-nine county offices 
of education out of 58 counties in California had 
teams attend the regional trainings, totaling 216 
attendees in all. Most teams had between three 
and ten members and included both county office 
and school district representatives. 


Like the master class, the regional trainings each 
lasted three full days and focused on the Paper 
Screen, and the facilitators followed the master 
class model they had experienced in November. 


9 See the full list at https://www.cde.ca.gov/ci/sc/im/adoptedsciprograms2018.asp. 
10 The S. D. Bechtel Jr. Foundation supported the California NGSS Collaborative in the development and delivery of the TIME 


master class and the two regional trainings as well as WestEd’s STEM Evaluation Unit in the evaluation of these events. 
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They had received binders containing instruc- Figure 6. Map of California counties’ 
tions, a facilitator script, and PowerPoint slides participation in December 2018 regional 
from the master class, and they had been asked TIME trainings 


to follow these as closely as possible. Each partici- 
pant of the regional trainings was also given these 
items, and it was impressed upon them that they, 
too, should closely follow the facilitator script and 
use the provided PowerPoint slides when they 
conducted TIME trainings themselves for their 


constituents. 


Local Trainings 


Throughout 2019, regional training attendees 
facilitated dozens of local TIME trainings. These 
facilitators, who were from county offices of educa- 
tion and school districts, both trained participants 


Key: 
= Did not attend regional training 


materials and guided participants to apply the @ = Attended regional training and did not host local training(s) 
@ = Attended regional training and hosted local training(s) 


in the TIME process using sample instructional 


TIME process to candidate instructional materials 


being considered for adoption.” 
*Out of the 39 county teams that hosted local 


Of the 49 counties with team attendance at the trainings, 36 of them were led by local training 
December 2018 regional trainings, 39 hosted local _ facilitators, and three of them were led by regional 
trainings, as shown in Figure 6. training facilitators — these three are indicated with 


asterisks in the map. 


11 Local TIME trainings have two primary purposes: (1) to lead participants through using the TIME toolkit and learning how 
to use the rubrics and identify evidence to assess the quality of sample instructional materials (through a three-day training) and 
(2) to guide participants through the process (generally two to three days) of facilitating school district staff as they use TIME to 
review and select candidate instructional materials for potential adoption. 


County office and district personnel who attended 


the regional trainings and were charged with 
leading their own TIME trainings back home 
anticipated challenges associated with conduct- 
ing TIME trainings. In interviews and surveys 
conducted soon after the regional trainings, find- 
ing time and meeting spaces for three full days 
was a tall order for many. Some said that it might 
be difficult to convince administrators of the value 
of such an investment. In addition, others were 
concerned about obtaining substitutes to enable 
teachers to participate in sessions held during the 
school year. 


While I completely see the relevance and 
professional development that is built in, 
communicating this with district super- 
intendents will be very important. I worry 
about the impact on students to pull 
teachers for this amount of time. So we will 
need to get creative about using the tool- 
kit in chunks and possibly offering incen- 
tives for teachers to meet in the summer. 
(County office of education participant in 


a regional TIME training) 


I feel that three days in a row like that is 
just too much. It’s a whirlwind, you come 
away shaking your head. There's just no 
way during school time that I think we can 
pull teachers out for three straight days... 
I don’t think it’s the money. I mean, it’s 
going to take a two- or three-day training, 


and to really be beneficial, you need teams 


from districts. So for some of our districts 
to be able to get five or six or seven subs, 
it's difficult. They just can't do it — 
they're in isolated areas. (County office of 
education participant in a regional TIME 


training) 


Other challenges mentioned by participants 
included associated monetary costs, such as print- 
ing materials and paying for substitutes. In addi- 
tion, one Early Implementer Core Teacher Leader 
attendee discussed how prior knowledge of the 
NGSS is helpful to be able to effectively apply the 
TIME process: 


My gosh, how are they going to evaluate 
materials when they don't really under- 
stand what the tool is asking them to look 
for? You can't find evidence [of aspects of 
NGSS in instructional materials] when 
you don't understand [the NGSS]. To 
me, it was sort of not a limitation of the 
presenters .... I just feel [the facilitators] 
were having to answer questions that 
went way beyond the tool. It was more like 
these people that were attending needed 
NGSS 101. 


Nonetheless, most TIME participants said that 
they saw value in following the three-day train- 
ing model and that they intended to try to do so. 
And although that Core Teacher Leader strongly 
believed that NGSS expertise is required for eval- 
uating instructional materials, she added that 
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the experience was useful to them nonetheless: “I the potential to reduce the value of a TIME train- 

think it helped them. It sort of put a spotlight on ing. Their response began with the assertion that, 

what they didn't know.” in general, districts should follow the Paper Screen 
process with fidelity. They also cautioned that 


certain variations can impact the integrity of the 


Va ri all ons n th e Pa pe r process and therefore the validity and reliability 

of the resulting information for making a recom- 
5 creen P fOCESs mendation to pilot test or adopt instructional 
During observations across the state, the evalua- materials. 


tion team saw several variations in the way local eee re 
Although some degree of variation is normal, the 

training facilitators implemented the Paper Screen ere . : 
variations to the prescribed TIME process outlined 


. Master class facilitat ked f 
process. Master class facilitators were asked for feraie eneald Ganaed 


their input on which variations they thought had 


Table 1. Variations observed by evaluators that can impact the integrity of the TIME process 


NEV ar-1 Ce) ale) o<-1aV/-1o Mo) Considerations 
evaluators 


Sessions were shorter than three All three days are necessary to build understanding 


days and to calibrate participants. 


Not all rubrics were used Participants cannot be fully trained and calibrated 
without using all four required rubrics. 


Each rubric measures unique and critical aspects of 
the instructional materials. 


Rubrics were combined There are purposeful redundancies built into the 
rubrics. Each rubric reviews the materials through a 
different, but coherent lens. 


Multiple passes through the materials also provide 
confirmation of the findings of the committee. 


Rubrics were addressed out of The rubrics build on one another. The materials have 
order to pass Rubric 1 before the next rubrics are considered. 
Rubric 4 is dependent on the other three rubrics. 


Groups did not have access to All components of the instructional materials are 

all components of candidate required for complete and valid scoring. 

instructional materials 

Process was rushed TIME takes time, so it’s recommended to allocate 
sufficient time to allow for contingencies that may 
impact the timing of the process. 


More than one grade/grade band Review should be focused on one grade/grade band and 
or chapter were examined at one chapter at a time. 
same time 
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During interviews, evaluators also learned of vari- 
ations that were used in other steps in the TIME 
process. For example, an interviewed participant 
noted that their district applied the Paper Screen 
and Pilot Materials steps of the TIME process 
concurrently, rather than using the Paper Screen 


to gather evidence about the presence of specific 


components indicating that the candidate mate- 
rial is appropriate to advance to the pilot step. By 
not conducting the Paper Screen first, it is possible 
that teachers used instructional materials in their 
classrooms that were poorly aligned to the NGSS 
and so were not suitable for the district's NGSS 


implementation efforts. 


1b 


The evaluation team presents the following 


recommendations to guide adoption commit- 


tees as they evaluate, select, and adopt science 


instructional materials to support districts’ NGSS 


implementation: 


> 


Identify the unique needs and priorities of 
your district that you hope instructional 
materials will help you address. Each district 
has particular needs that affect its instruc- 
tional priorities. For example, a district's 
vision for instruction is often shaped by the 
resources it has, as well as by its teacher 

and student populations. It is important for 
adoption committees to evaluate instructional 
materials based on these unique needs in order 
to select the best materials possible. 


Acknowledge the benefit of rigorously review- 
ing candidate instructional materials in all 
content areas. Adoption represents a hefty 
financial investment and a commitment of 10 
or more years to the adopted product. Districts 
should realize how critical it is to carefully 
analyze and vet candidate instructional 
materials before making such an investment. 
A thorough review could also reveal potential 
gaps in the materials and inconsistencies 
across grade levels. 


Consider taking advantage of TIME, an 
existing tool developed specifically for review 
of NGSS-aligned instructional materials. 
Even though many attendees pointed out that 
implementing the TIME process is demand- 
ing and challenging, they acknowledged the 
value of using the TIME process to choose 
appropriate instructional materials to match 
district needs and priorities. To participate 
ina TIME training yourself, please contact a 
member of the California NGSS Collaborative 


(the California Department of Education, 

the California Science Project, the California 
Science Teachers Association, the California 
County Superintendents Educational Services 
Association and its Curriculum Instruction 
Science Committee, local county offices of 
education, and the K-12 Alliance). 


Consider the TIME approach as a support for 
learning about the NGSS. Because the NGSS 
are so different from past science standards, 
teachers will need deep professional learning 
to understand how to teach them as intended. 
Going through the TIME process not only helps 
participants evaluate science instructional 
materials, but also helps teachers become more 
familiar with the standards themselves. 


Know that even after adopting NGSS instruc- 
tional materials, teachers and administrators 
will continue to need professional learning. 
Once your district has adopted instructional 
materials, do not consider your district 
finished with NGSS implementation. High- 
quality instructional materials are a corner- 
stone to launching NGSS implementation 
efforts, but ongoing professional learning is 
needed to support teachers in how to best 
use those materials to support students’ 
three-dimensional learning. Administrators 
need to learn what NGSS instruction should 
look like in order to support teachers in best 
using the materials. 
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Data collected for this report came from three 
phases of TIME trainings. A small group of 
regional leaders were trained in the TIME process 
during the master class. These regional leaders, 
in turn, facilitated two regional trainings — 

one in Northern California and one in Southern 
California. The attendees of the regional train- 
ings then returned to their districts and counties 


all over the state, where they conducted local 


trainings. 


There were two time points of data collection 


related to the local trainings — data collected 


shortly after participants attended trainings and 


data collected months after these trainings. A 


couple of Early Implementer teachers participated 


in these TIME-specific interviews. 


Table Ai. Dates of data collection, number of participants, and sources of data 


Phase of TIME Date(s) el ad(el]ey-lal ey Data collected 
ive Tialtale hs 


Master class November 2018 
Regional trainings December 2018 (two 
in Northern and regional trainings) 


Southern California 


January—February 
2019 (interviews) 


23 regional leaders 


206 attendees (178 
completed the 
survey) 


Four facilitators 
and seven attendees 
interviewed 


Sign-in sheets 
Agenda 


Observation notes 


Sign-in sheets 
Agenda 
Observation notes 


Attendee survey 
responses 


Attendee and 
facilitator interviews 
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Phase of TIME Participants DEL t= exe} |(=oi ¢-1o| 
ide-Tlaltale ks 


Local trainings in January—October 41 facilitators and Observation notes 


counties and school 2019 (surveys for 425 attendees 
Attendee and 


districts local training completed surveys 
facilitator survey 


participants and 


Feciiineons| 16 facilitators responses 


and 19 attendees 


April—-October 2019 interviewed SAREE EEN 


facilitator interviews 


(observations) 
May-October 2019 (Data collected 
(interviews) shortly after 
trainings) 
Local trainings in December 2019- Seven facilitators Observation notes 
counties and school January 2020 and eight attendees 
fae : Attendee and 
districts (follow-up interviews interviewed 


with local/training facilitator interviews 


facilitators and (Data collected 
attendees) months after 


trainings) 


In addition, data were collected from Early Implementer leaders, as shown in Table A2. 


Table A2. Dates of data collection, number of participants, and sources of data related to 
Early Implementer leaders 


bate(:| een | [Faiticlpants paesesieeres 


January 2020 (Master class Three master class Master class facilitator focus 
facilitator focus group, facilitators participated ina group 


Early Implementer Initiative focus group 


: F : : Project Director interviews 
Project Director interviews, 


Apeaional Direct Eight Project Directors 
SUMS) NEG HeiniaN CMNSCWwer ; : Regional Director interviews 
: interviewed 

interviews) 


Three Regional Directors 


interviewed 
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Appendix B. Paper Screen 
Rubrics 


The following are Paper Screen rubrics that are used to determine an overall score for the instructional 
materials being evaluated. Each criterion within these rubrics has its own guidelines that are individ- 
ually applied to the instructional materials and scored before participants decide on an overall score 
for the rubric. Scores range from 1 (Low Quality) to 5 (High Quality). If a set of instructional materials 
is given an overall score of 1 on Rubric 1, Foundations, the review process does not continue because 
the quality of the materials is not high enough to warrant further review. A set of materials earning a 
score of 3 or 5 on Rubric 1 undergoes scoring with the remaining required rubrics (i.e., Student Work, 
Monitoring Student Progress, and Teacher Support). The fifth rubric (Program Evaluation) is optional 


and may be applied before the Pilot Materials step. 


Figure B1. CA NGSS TIME Paper Screen Rubric 1: Foundations 


Components and Indicators 


F1. Presence of Phenomena/Problem. The materials 
include phenomena/problems that 

¢ have the potential to drive student learning. 

« have the potential to relate across the dimensions. 


F2. Presence of Three Dimensions. The materials include 
the three dimensions, such that 
« the DCls, SEPs, and CCCs are present and have the 
potential to support student learning. 
when engineering design is a learning focus, it is 
integrated with the appropriate dimensions (i.e., 
engineering is not isolated). 
F3. Presence of Environmental Principles and Concepts 
(EP&Cs). The materials include (as applicable) 
instructional content that incorporates the California 
EP&Cs. 
opportunities for students to examine the interactions 
and interdependence of human societies and natural 
systems. 
opportunities for students to develop and/or implement 
solutions to real-world environmental problems. 
F4. Presence of Logical Sequence. Materials demonstrate 
appropriate sequencing of three dimensions when 
they include a targeted set of DCIs, SEPs, and CCCs 
within a sequence; 
the sequence is clear and logical across the DCls; and 
the SEPs and CCCs are potentially sufficient and 
appropriate for students to figure out the phenomena or 
problems. 


phenomenon or problems are linked to each other. 


High Quality 
5 


The materials include 
phenomena/problems 
that have strong 
potential to drive 
student leaming toward 
the targeted learning 
goals. 


The materials 
consistently provide 
opportunities for 
students to develop and 
use grade-appropriate 
elements of the three 
dimensions. 


The materials 
consistently provide 
opportunities for 
students to examine 
and use elements of the 
EP&Cs. 


The materials 
consistently exhibit a 
clear, logical, and 
appropriate sequence 
across the three 
dimensions. 


Section 3: Paper Screen Rubric 1 — H7 
Designed for the NGSS: Foundations Rubric 


Medium Quality 
3 


The materials include 
phenomena/problems 
that have some 
potential to drive 
student leaming toward 
the targeted leaming 
goals. 


The materials 
occasionally provide 
opportunities for 
students to develop and 
use grade-appropriate 
elements of the three 
dimensions. 


The materials 
occasionally provide 
opportunities for 
students to examine 
and use elements of the 
EP&Cs. 


The materials 
occasionally exhibit a 
clear, logical, and 
appropriate sequence 
across the three 
dimensions. 


Low Quality 
1 


The materials include 
phenomena/problems 
that have limited 
potential to drive 
student learning toward 
the targeted leaming 
goals. 


The materials rarely 
provide opportunities for 
students to use grade- 
appropriate elements of 
the three dimensions. 


The materials rarely 
provide opportunities for 
students to examine 
and use elements of the 
EP&Cs. 


The materials rarely 
exhibit a clear, logical, 
and appropriate 
sequence across the 
three dimensions. 


Source: Adapted from BSCS Science Learning, developed in collaboration with the K-12 Alliance at WestEd and 


Achieve, Inc. 
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Figure B2. CA NGSS TIME Paper Screen Rubric 2: Student Work 


Section 3: 


Designed for the NGSS: Student Work Rubric 
High Quality 
5 


Designed for CA NGSS: Student Work Rubric 


SW1. Phenomena/Problems. Materials provide anchoring and 
investigative phenomena/problems that: 


engage students as directly as possible in authentic and relevant 
experiences. 


are matched to targeted learning goals. 


can be figured out/solved using scientifically accurate 
understandings and abilities. 


make connections beyond and to their daily lives including 
to their homes, neighborhoods, communities, local 
environment, and/or cultures. 


SW2. Three-dimensional Conceptual Framework. Materials 
include learning experiences that help students to build scientifically 
accurate understandings and abilities through opportunities for 
students to: 


link prior knowledge to negotiated new understanding and abilities. 


do work that approximates the nature of science. 


use reasoning to connect grade-appropriate SEP, DCI, and CCC 
elements and EP&C’s (when applicable). 


ask and answer questions that link learning over time. 


negoliate new understandings and abilities by comparing their 
ideas, their peers’ ideas, and ideas encountered in the learning 
experience(s). 


their understandings and abilities in a variety of ways. 


Materials consistently 
offer quality 
phenomena/ 
problems sufficient to 
motivate and drive 
student learning. 


Materials 
consistently include 
learning 
experiences that 
help students build 
from prior 
experiences to 
negotiate new 
understandings and 
abilities and apply 
their understandings 
in a variety of ways. 


Medium Quality 
3 


Materials sometimes 
offer quality 
phenomena/ problems 
sufficient to motivate 
and drive student 
learning. 


Materials sometimes 
include learning 
experiences that help 
students build from 
prior experiences to 
negoliale new 
understandings and 
abilities and apply 
their understandings 
in a variety of ways. 


Paper Screen Rubric 2—H14 


Low Quality 
1 


Materials rarely 
offer quality 
phenomena/ 
problems sufficient 
to motivate and 
drive student 
learning. 


Materials rarely 
include learning 
experiences that help 
students build from 
prior experiences to 
negoliale new 
understandings and 
abilities and apply 
their understandings 
in a variety of ways. 
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Designed for CA NGSS: Student Work Rubric 


SW3. Prior Knowledge. Materials leverage students’ prior 
knowledge and experiences to motivate student learning in ways that: 


® make visible students’ prior knowledge and experiences related 
to the anchoring and investigative phenomena/problems and 
relevant SEPs, DCls, and CCCs and EP&Cs (when applicable). 


revisil students’ early ideas to see how they have changed (or 
nol) as they figure oul phenomena/solve problems. 


make explicit links to new ideas and practices being developed 

by students. 
SW4. Metacognitive Abilities. Materials include learning experiences 
for students to: 


set and monitor their learning in light of the targeted learning goals. 


consider, over time, what and how they have learned 
across the three dimensions. 


articulate how the three dimensions helped them figure out anchor 
and investigative phenomena/solve problems. 


SWS5. Equitable Learning Opportunities: Materials ensure that all 

students, including those from non-dominant groups and with diverse 

learning needs, have access to the targeted learning goals and 

experiences, including: 

® appropriate reading, writing, listening, and/or speaking 
alternatives for students who are English language learners, have 


special needs, read below the grade level, or have high interest 
and have already met the intended learning goals. 


culturally-relevant contexts and examples that support all students. 


opportunities to cultivate interest and confidence as scientists and 
engineers for all students. 


High Quality 
5 


Materials 
consistently 
leverage student 
prior knowledge 
and experiences lo 
motivate their 
learning. 


The materials 
provide students 
with regular, explicit 
opportunities to 
consider how their 
learning 
experiences 
changed their 
thinking. 


Mast learning 
experiences in 
materials are 
designed such that 
students can 
engage 
meaningfully in a 
variety of ways, with 
multiple access 
points, and with 
supports for 
students. 


Section 3: Paper Screen Rubric 2 -H14 


Medium Quality 
3 


Materials sometimes 
leverage student prior 
knowledge and 
experiences lo 
motivate their 
learning. 


The materials provide 
students with some 
opportunities to 
consider how their 
learning experiences 


changed their thinking. 


Some learning 
experiences in 
materials are 
designed such that 
students can engage 
meaningfully in a 
variety of ways, with 
multiple access 
points, and with 
supports for students 


Low Quality 
1 


Materials rarely 
leverage student prior 
knowledge and 
experiences and when 
included, they do not 
relate lo the 
phenomena or 


problems. 


The materials provide 
few opportunities for 
students to consider 
how their learning 
experiences changed 
their thinking. 


Few leaming 
experiences in 
materials are 
designed such that 
students can engage 
meaningfully in a 
variety of ways, with 
multiple access 
points, and with 
supports for students. 


Source: Adapted from BSCS Science Learning, developed in collaboration with the K-12 Alliance at WestEd and 


Achieve, Inc. 
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Figure B3. CA NGSS TIME Paper Screen Rubric 3: Monitoring Student Progress 


SP1. Monitoring Three-Dimensional Learning and EP&Cs 


Section 3: Paper Screen Rubric 3 —H19 


Designed for CA NGSS: Monitoring Student Progress Rubric 


If Applicable. Assessments are designed to 


SP2. Capturing Student Progress. The assessments within 


ensure that students use SEPs integrated with DCls and 
CCCs to demonstrate their understanding of phenomena 
and/or design solutions to problems. 


connect student learning experiences to the targeted 
learning goals. 


elicit observable evidence of students’ knowledge of and 
ability to use grade-level appropriate elements of the 
three-dimensions. 


ensure that students use EP&Cs where applicable to 
demonstrate their understanding of environmental 
phenomenon/problem solution. 


a unit: 


include pre-, formative, summative, and self- or peer- 
assessment measures thal assess three-dimensional 
learning; and 


these different lypes of measures are connected to one 
another to demonstrate student progress over time. 


Assessments are 


consistently designed to 
connect to learning goals 
and prompt students to 
apply appropriate 
elements of the three 
dimensions and EP&Cs 
where applicable to 
demonstrate their 
understanding of the 
phenomenon/problem 
solution. 


There are multiple 
opportunities, using 
more than one type of 
measure, to 
demonstrate learning 
and these measures are 
strongly connected to 
show student progress 
both in and across the 
three dimensions. 


Assessments are 
sometimes designed to 
connect lo learning 
goals and prompt 
students to apply 
elements of two or three 
dimensions and EP&Cs 
where applicable to 
demonstrate their 
understanding of the 
phenomenon/problem 
solution. 


There are multiple 
opportunities, using 
more than one type of 
measure, to 
demonstrate learning 
and these measures are 
connected to show 
student progress both in 
and across the three 
dimensions. 


Assessments are either 
not connected to 
learning goals and/or 
they prompt students to 
apply elements of only 
one dimension to 
demonstrate their 
understanding of the 
phenomenon/problem 
solution. EP&Cs are not 
included, even where 
applicable. 


There is not an 
appropriate combination 
of measures for students 
to demonstrate progress 
both in and across the 
three dimensions. 


lal 
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SP3. Variety of Measures. Assessments are matched to 

targeted learning goals and elicil a full range of student 

thinking by 

® providing clear expectations (e.g., rubric) to students so 
they understand how they can demonstrate their 
knowledge. 


using a variely of measures (e.g., performance tasks, 
discussion questions, constructed response questions, 
project- or problem-based tasks, portfolios, and justified 
multiple choice). 


providing set(s) of tasks so that students can demonstrate 
their understanding of the same learning goals in multiple 


ways. 
SP4. Equitable Access. Assessments are designed to be 


¢ free from bias (e.g., gender, racial, socioeconomic status, 
cultural). 


accessible to all students (e.g., reading level, 
accommodations). 
SP5. Use of Assessment. 


The materials provide self- or peer-assessments that allow 
students to reflect on and monitor their learning over time. 


The assessments 
include a wide variely of 
formats with clear 
expectations that allow 
students to demonstrate 
their understanding of 
the learning goals in 
multiple ways. 


Most assessments are 
free from bias and are 
accessible. 


The materials miss few 
or no opportunities to 
encourage self- or peer- 
assessments that allow 
students to reflect on 
and monitor their 


progress. 


Section 3: Paper Screen Rubric 3—H19 


The assessments 
include some variety of 
formats with clear 
expectations that allow 
students to demonstrate 
their understanding of 
the learning goals in 
multiple ways. 


Some assessments are 
free from bias and are 
accessible. 


The materials miss 
some opportunities to 
encourage self- or peer- 
assessments that allow 
students to reflect on 
and monitor their 


progress. 


The assessments use 
just one format and/or 
the expectations for 
students to demonstrate 
their knowledge are 
absent or unclear. 


Few assessments are 
free from bias and are 
accessible. 


The materials miss all or 
most opportunities to 
encourage self- and 
peer-assessments that 
allow students to reflect 
on and monitor their 


progress. 


Source: Adapted from BSCS Science Learning, developed in collaboration with the K-12 Alliance at WestEd and 


Achieve, Inc. 
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Figure B4. CA NGSS TIME Paper Screen Rubric 4: Teacher Support 


Section 3: Paper Screen Rubric 4 —- H22 


Designed for CA NGSS: Teacher Support Rubric 


Designed for CA NGSS: Teacher Support Rubric 


TS1. Phenomena/Problems Driven Three-Dimensional Learning. 
Teacher materials provide 


@ background information about the phenomena or problems included 
in the leaming sequence and across sequences. 


an explanation of the role of phenomena or problems in driving 
student learning. 


rationale for why the unit phenomena or problems were selected for 
the targeted DCIs, SEPs, CCCs, and EP&Cs (when applicable). 


Evidence found in: F1, F2, SW1, SW2, SP1 


TS2. Coherence. Teacher materials describe and provide a rationale for 


e@ the conceptual framework and sequence of ideas, practices, and 
learning experiences in the learning sequences and across 
sequences. 


strategies for linking student experiences across lessons to ensure 
student sense-making and/or problem-solving focused on 
phenomena or problems is linked to learning across all three 
dimensions. 


connections to other science domains, nalure of science, 
engineering, technology, and applications of science, math, ELA, and 
EP&Cs (when applicable). 


Evidence found in: F2, F3, F4, SW2, SP2 


High Quality 
5 


Materials provide 
clear guidance lo 
teachers on how 
students develop, 
use, and integrate the 
three dimensions and 
EP&Cs (where 
appropriate) to make 
sense of phenomena 
or design solutions to 
problems. 


Materials provide 
strong support for 
understanding unit 
coherence and 
helping students link 
experiences to 
learning across all 
three dimensions and 
to phenomena or 
problems. 


Medium Quality 
3 


Materials provide 
some guidance to 
teachers about how 
students develop, 
use, and integrale the 
three dimensions and 
EP&Cs (where 
appropriate) to make 
sense of phenomena 
or design solutions to 
problems. 


Materials provide 
some support for 
understanding unit 
coherence and 
helping students link 
experiences to 
learning across all 
three dimensions and 
to phenomena or 


problems. 


Materials provide little 
guidance to teachers 
about how students 
develop, use, and 
integrate the three 
dimensions and 
EP&Cs (where 
appropriate) to make 
sense of phenomena 
or design solutions to 
problems 


Materials provide little 
support for 
understanding unit 
coherence and 
helping students link 
experiences to 
learning across all 
three dimensions and 
to phenomena or 


problems. 
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Designed for CA NGSS: Teacher Support Rubric 


TS1. Phenomena/Problems Driven Three-Dimensional Learning. 
Teacher materials provide 


@ background information about the phenomena or problems included 
in the leaming sequence and across sequences. 


an explanation of the role of phenomena or problems in driving 
student learning. 


rationale for why the unit phenomena or problems were selected for 
the targeted DCIs, SEPs, CCCs, and EP&Cs (when applicable). 


Evidence found in: F1, F2, SW1, SW2, SP1 


TS2. Coherence. Teacher materials describe and provide a rationale for 


e the conceptual framework and sequence of ideas, practices, and 
learning experiences in the learning sequences and across 
sequences. 


strategies for linking student experiences across lessons to ensure 
student sense-making and/or problem-solving focused on 
phenomena or problems is linked to learning across all three 
dimensions. 


connections to other science domains, nature of science, 
engineering, technology, and applications of science, math, ELA, and 
EP&Cs (when applicable). 


Evidence found in: F2, F3, F4, SW2, SP2 


High Quality 
5 


Materials provide 
clear guidance to 
teachers on how 
students develop, 
use, and integrate the 
three dimensions and 
EP&Cs (where 
appropriate) to make 
sense of phenomena 
or design solutions to 
problems. 


Materials provide 
strong support for 
understanding unit 
coherence and 
helping students link 
experiences lo 
learning across all 
three dimensions and 
to phenomena or 
problems. 


Section 3: Paper Screen Rubric 4 —- H22 


Designed for CA NGSS: Teacher Support Rubric 


Medium Quality 
3 


Materials provide 
some guidance to 
teachers about how 
students develop, 
use, and integrate the 
three dimensions and 
EP&Cs (where 
appropriate) to make 
sense of phenomena 
or design solutions to 
problems. 


Materials provide 
some support for 
understanding unit 
coherence and 
helping students link 
experiences lo 
learning across all 
three dimensions and 
to phenomena or 
problems. 


Materials provide little 
guidance to teachers 
about how students 
develop, use, and 
integrate the three 
dimensions and 
EP&Cs (where 
appropriate) to make 
sense of phenomena 
or design solutions to 
problems 


Materials provide little 
support for 
understanding unit 
coherence and 
helping students link 
experiences to 
learning across all 
three dimensions and 
to phenomena or 
problems. 


Source: Adapted from BSCS Science Learning, developed in collaboration with the K-12 Alliance at WestEd and 


Achieve, Inc. 
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Figure B5. CA NGSS TIME Paper Screen Rubric 5: Program Evaluation 


Designed for CA NGSS: Program Evaluation Rubric 


Designed for CA NGSS: Program Evaluation Rubric 


PE1. Progressions of Learning. Within a program, learning experiences are more 
likely to help students develop a greater sophistication of understanding of the 
elements of SEPs, CCCs, and DCls, and EP&Cs (when applicable) when teacher 
materials: 


e make it clear how each of the three-dimensions builds logically and 
progressively over the course of the program and make clear how 
students 


engage in the science and engineering practices with increasing 
grade-level appropriate complexity over the course of the program. 


utilize the crosscutting concepts with increasing grade-level 
appropriate complexity over the course of the program. 


engage in grade level/band appropriate disciplinary core ideas. 


teacher materials make clear how the performance expectations are 
addressed in the program. 


provide a rationale for a logical sequence/treatment of Engineering, 
EP&Cs, and the Nature of Science (NoS). 


PE2. Unit-to-Unit Coherence. Unils across a program demonstrate 
coherence when student materials: 


e are designed with an appropriale sequence and development of DCls, 
GGCs, and SEPs to support students in demonstrating learning across 
a program as they figure out phenomena/problems. 


make explicit connections from one unit to the next across the three- 
dimensions to connect prior learning, current learning, and future 
learning as they figure oul phenomena/problems. 


support students in making connections across units and disciplines by 
helping student negotiate more sophisticated understandings and 
abilities. 


High Quality 
5 


Materials enact 
progressions of 
learning that 
have all or most 
of the quality 
characteristics 


The materials 
consistently 
justity 
sequencing and 
demonstrate 
strong unit-to-unit 
coherence for 
developing 
competence in 
three- 
dimensions. 


Medium Quality 
Ky 


Materials enact 
progressions of 
learning that have 
some of the 
quality 
characteristics 


The materials 
occasionally justify 
sequencing and 
sometimes 
demonstrate 
strong unit-to-unit 
coherence for 
developing 
competence in 
three-dimensions. 


Section 3: Paper Screen Rubric 5 —- H28 


Low Quality 
1 


Materials enact 
progressions of 
learning that have 
none or few of the 
quality 
characteristics 


The materials 
never justify 
sequencing and 
rarely demonstrate 
unit-to-unit 
coherence for 
developing 
competence in 
three-dimensions. 


$1 
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Designed for CA NGSS: Program Evaluation Rubric 


PE3. Program Assessment System. Over the course of the program, teacher 
materials demonstrate a system of assessments that 


« coordinates the variely of ways student learning is monitored to provide 
information to students and teachers regarding student progress for all 
three-dimensions of the standards and toward proficiency at the identified 


grade-level/band performance expectations. 


includes support for teachers and other leaders to make program-level 
decisions based on unit, interim, and/or year-long summative assessment 
data. 

is driven by an assessment framework and provides a structured 


conceptual map of student learning along with details of how achievement 
of the outcomes can be measured. 


High Quality 
5 


The materials 
use a program- 
level assessment 
system that has 
all or most of the 
quality 
characteristics 


Section 3: Paper Screen Rubric 5 - H28 
Medium Quality Low Quality 
3 1 


The materials use | The materials 

a program-level use a program- 
assessment level assessment 
system that has system that has 
some of the few or none of 
quality the quality 
characteristics characteristics 


Source: Adapted from BSCS Science Learning, developed in collaboration with the K-12 Alliance at WestEd and 


Achieve, Inc. 


Core Leadership Team (CLT) — Group of three to five administrators and five to eight 
teachers established at each district at the beginning of the Initiative. The CLT meets 
with their Project Director regularly during each school year to plan and lead all Early 


Implementers Initiative activities. They meet with their K-12 Alliance Regional Director 
for six Technical Assistance Days each school year. 


Core Teacher Leader (CTL) — Teacher member of the Core Leadership Team. Provides 
professional learning to Teacher Leaders, other teachers, and/or administrators in their 
district or at projectwide events such as the Summer Institute. 


Crosscutting Concepts (CCCs) — One of the three NGSS dimensions and a way of link- 
ing the different domains of science. CCCs include patterns; cause and effect; scale, 
proportion, and quantity; systems and system models; energy and matter; structure and 
function; and stability and change. 


Dimensions of the NGSS — The NGSS includes three dimensions: Disciplinary Core 
Ideas (what scientists know), Crosscutting Concepts (how scientists make connections 
among the sciences), and Science and Engineering Practices (what scientists and engi- 
neers do and how scientific knowledge develops). 


Disciplinary Core Ideas (DCIs) — One of the three NGSS dimensions. According to 
National Research Council's Framework for K-12 Science Education, disciplinary core 
ideas are the important concepts in each of four domains: physical sciences; life sciences; 
Earth and space sciences; and engineering, technology, and applications of science. 


Environmental Principles and Concepts (EP&Cs) — Today, the most common schema 
for the field of environmental literacy in California is the Environmental Principles 

and Concepts. The five Principles of the EP&Cs are the following: People Depend on 
Natural Systems; People Influence Natural Systems; Natural Systems Change in Ways 
That People Benefit From and Can Influence; There Are No Permanent or Impermeable 
Boundaries That Prevent Matter From Flowing Between Systems; and Decisions 
Affecting Resources and Natural Systems Are Complex and Involve Many Factors. Each 
of the five Principles contains several Concepts. 


K-8 NGSS Early Implementers Initiative — Six-year initiative (Summer 2014 through 
spring 2020) supporting implementation of the NGSS by eight public school districts and 
two charter management organizations in California. Developed by the K-12 Alliance 

at WestEd in collaboration with the California State Board of Education, the California 
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Department of Education, and Achieve, the Early Implementers Initiative builds the 
capacity of participating local education agencies to fully implement the NGSS in 
grades K-8. 


K-12 Alliance — A WestEd program of science education leaders and professional learn- 
ing providers who plan and deliver all projectwide activities for the Early Implementers 
Initiative. 


NGSS — A set of K-12 science content standards developed by states to improve science 
education for all students. They are composed of three dimensions based on the National 
Research Council's Framework for K-12 Science Education. 


Phenomena — Phenomena are occurrences in the natural or built world that cause us to 
wonder and ask questions. There are two types of phenomena, anchoring and investigative. 


Project Director — District person responsible for leading all Early Implementers 
Initiative activities for the district and representing the district at monthly Initiative- 
wide planning meetings with Regional Directors. 


Science and Engineering Practices (SEPs) — One of the three NGSS dimensions, SEPs 
are the behaviors that scientists engage in as they investigate and build models and 
theories about the natural world and the key set of engineering practices that engineers 
use as they design and build models and systems. SEPs include asking questions (for 
science) and defining problems (for engineering); developing and using models; planning 
and carrying out investigations; analyzing and interpreting data; using mathematics 
and computational thinking; constructing explanations (for science) and designing solu- 
tions (for engineering); engaging in argument from evidence; and obtaining, evaluating, 
and communicating information. 


Teacher Leader (TL) — One of 30 to 70 teachers in each district who joined the Early 
Implementers Initiative in year 2, one year after the CLTs joined. Teacher Leaders attend 
annual Summer Institutes and participate in two Teacher Learning Collaboratives each 
school year (one in the fall and one in the spring) and other district-level professional 
learning. 


Teaching Learning Collaborative (TLC) — Lesson study activity in years 1 through 

4 of the Early Implementers Initiative. Each TLC brings together three to four same- 
grade Early Implementers Initiative teachers from different schools within the district. 
Teachers plan and teach a lesson to two classrooms of students and debrief after each 
lesson is taught, during which they examine student work from the lesson. Each TL 
participates in two TLCs per year. 


Toolkit for Instructional Materials Evaluation (TIME) — A suite of tools and processes 
for curriculum-based professional learning that is designed to help educators evaluate, 
select, and implement instructional materials aligned to the NGSS. 
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